Always-available input through finger instrumentation

ABSTRACT

A finger device initiates actions on a computer system when placed in contact with a surface. The finger device includes instrumentation that captures images and gestures. When in contact with a surface, the finger device captures images of the surface and gestures made on the surface. The finger device also transmits the images and gesture data to the computer system. An application on the computer system matches the images received from the finger device to a representation of the surface, identifies an action associated with the surface representation and gesture, and executes the action. Instrumenting the finger instead of the surface, allows a user to configure virtually any surface to accept touch input.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a divisional of the co-pending U.S. patentapplication titled, “ALWAYS-AVAILABLE INPUT THROUGH FINGERINSTRUMENTATION,” filed on Oct. 2, 2013 and having application Ser. No.14/044,678, which claims priority benefit of the U.S. Provisional PatentApplication titled, “MAGIC FINGER ALWAYS AVAILABLE INPUT THROUGH FINGERINSTRUMENTATION,” filed on Oct. 2, 2012 and having Application No.61/708,790. The subject matter of these related applications is herebyincorporated herein by reference.

BACKGROUND OF THE INVENTION Field of the Invention

Embodiments of the present invention generally relate to computer inputdevices. More specifically, embodiments presented herein disclose afinger device which allows virtually unlimited interaction with anysurface.

Description of the Related Art

Many electronic devices (e.g. smartphones and tablet computers) usetouch screens as a mechanism for user input. For instance, tabletcomputers include touchscreens that accept touch input. Tablet computersperform various tasks in response to different gestures and touches. Atablet computer may interpret a swipe on the touchscreen as a command toscroll though a screen. Likewise, a tablet computer may interpret a tapon the touchscreen as a command to open an application.

Surfaces that accept touch input (e.g. touchscreens) rely oninstrumentation to detect touch input. Typically, the surface includesan array of sensors that detect where a finger is contacting thesurface. Sensors, such as cameras, may also be placed proximate to asurface to detect how a user touches the surface. Instrumenting asurface to accept touch input can be costly and complex, which limitsthe number of surfaces that accept touch input.

SUMMARY OF THE INVENTION

One embodiment of the invention includes a method for initiating anaction in response to a user touching a surface. This method maygenerally include receiving an image from a device instrumenting afinger of the user. The finger is in contact with a surface. This methodmay also include identifying, from the image, the surface contacted bythe finger of the user and matching the identified surface to an actionexecuted by a computing device. This method may also include executingthe action.

Another embodiment of the invention includes a device worn on a finger.The device itself may comprise a camera configured to capture images ofa surface and a microcontroller configured to detect when the finger ofa user wearing the device touches a surface. In response to the figuretouching a surface, the device may (i) capture an image of the surfaceand (ii) transmit the image to a computing system.

Other embodiments include, without limitation, a computer-readablemedium that includes instructions that enable a processing unit toimplement one or more aspects of the disclosed methods as well as asystem having a processor, memory, and application programs configuredto implement one or more aspects of the disclosed methods. One advantageof the disclosed techniques is that the user is able to interact with avariety of surfaces without having to instrument the surfaces.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the inventioncan be understood in detail, a more particular description of theinvention, briefly summarized above, may be had by reference toembodiments, some of which are illustrated in the appended drawings. Itis to be noted, however, that the appended drawings illustrate onlytypical embodiments of this invention and are therefore not to beconsidered limiting of its scope, for the invention may admit to otherequally effective embodiments.

FIG. 1 illustrates a system configured to respond to a user touching asurface with a instrumented finger device, according to one embodiment.

FIG. 2 illustrates an example of a device used to instrument a finger,according to one embodiment.

FIG. 3 illustrates regions of an image of a surface, according to oneembodiment.

FIG. 4 illustrates a method for determining when a finger device iscontacting a surface, according to one embodiment.

FIG. 5 illustrates a method for associating an action to surface touchesmade by a user wearing an instrumented figure device, according to oneembodiment of the present invention.

FIG. 6 illustrates a method for initiating an action based on a usertouching a surface with an instrumented finger device, according to oneembodiment.

FIG. 7 illustrates a computing system configured to implement one ormore aspects of the present invention.

DETAILED DESCRIPTION

Embodiments presented herein provide an instrumented device that cansense and discriminate surfaces touched by an individual wearing thedevice. The instrumented finger device can be used to initiate actionsin response to a user touching a surface. In one embodiment, a user mayinitiate an action by performing a gesture (e.g. tapping, swiping, orpressing) with a finger on a surface. The user can interact withvirtually unlimited types of surfaces. That is, rather thaninstrumenting an object to receive user input via a touch-sensitivedisplay (or other instrumented surface), the user instruments theirfinger. Once instrumented, the device worn by the user senses what isbeing touched and initiates actions in response. Doing so allowsvirtually any given surface to be used as a trigger for some action.Accordingly, in one embodiment, a user wears a device on their finger.When the user touches a surface, the device contacts the surface. Thedevice may include sensors that capture images of the surface touched bythe user. When the user touches a surface, the device transmits imagesof the surface to a computer system. An application running on thecomputer system receives input from the device and executes an action inresponse. Significantly, the action depends on what surface is touchedby the user.

To execute an action, the application matches images received from thedevice to a library. If a match is found, the application determineswhat action has been associated with the touched surface and executesthat action. Thus, the device inverts the typical relationship betweenthe finger and touch input on a surface, i.e. the device instruments thefinger instead of the surface.

To configure a surface to accept touch input, the application maps thesurface to an action. The application presents an interface that allowsa user to map actions to various combinations of surfaces and gestures.

For example, the user could configure the application to answer a phonecall (or mute a ringtone) when the user touches a finger on a specificregion on a shirt. When the user touches the shirt, the device capturesimages of the shirt. The device transmits the images captured when thedevice is touched against a surface to the application. Continuing withthe example above, the application matches the surface of a shirt withan action to answer a call (or mute a ringtone) or to direct a call tovoicemail. For example, a user could configure the application to answera call if they tap their shirt, but send a call to voicemail if they taptheir pants.

Further, the device may be able to sense gestures made by a user—basedon dynamic changes in the image captured by the device. In such as case,the device transmits data representing a gesture along with images ofthe surface to the computer system. The application then determines anaction associated with the touched surface and the gesture made on thesurface.

In the following description, numerous specific details are set forth toprovide a more thorough understanding of the present invention. However,it will be apparent to one of skill in the art that the presentinvention may be practiced without one or more of these specificdetails. In other instances, well-known features have not been describedin order to avoid obscuring the present invention.

FIG. 1 illustrates a system configured to respond to a user touching asurface with an instrumented finger device, according to one embodiment.As shown, the system 100 includes a device 120 worn on finger 115 andcoupled to a computer system 110. The computer system 110 may be apersonal computer, laptop, or a mobile device, e.g. tablet, smart phone,smart watch, or headset. Illustratively, the computer system 110includes a touch controller 112. The device 120 is coupled to the touchcontroller 112 via a communications link 105.

To detect the user touching a surface, the device 120 includes a camerathat captures images. That is, when the user contacts a surface, thecamera captures images of the surface. When the user touches a surface,the device 120 transmits images of the touched surface to the touchcontroller 112, via communications link 105. Communications link 105 maybe a wired connection that transports data and power, such as a USBconnection. In other embodiments, the communications link 105 may be awireless connection that transports data, but not power, such as aBluetooth connection.

The touch controller 112 is an application configured to execute actionsin response to a user touching a surface. That is, the touch controller112 executes an action associated with a touched surface. The user mapsactions to surfaces using the touch controller 112. Actions may includecommands to launch applications or send notifications, or a variety ofother programmatic responses to a user touching a given surface.

In one embodiment, the computer system 110 includes a library thatstores representations of surfaces. A surface may be the natural surfaceof a particular object or a surface created by a user. For example, auser might print a series of text characters with a small font sizewhile varying the color or darkness of the characters to create patternsor icons. Other markers could be miniature bar codes, dot codes,fiducial markers, etc. For instance, the user could create an icon torepresent desktop applications, such as a word processor, browser,spreadsheet, email client, etc. In such a case, the user could affix theprinted markers to a surface at their desk. When the user then touchesone of the printed markers with the instrumented finger, the touchcontroller 112 could determine which printed marker the user touched andlaunch the corresponding application. Similarly, functions of anapplication could be associated with printed markers or with differentsurfaces available for a user to touch.

The stored representations of each surface may include a label, a sampleimage, features of the surface, and mappings to various actions. Personsskilled in the art will recognize that a variety of techniques may beused to extract features of a surface shown in an image. For example, inone embodiment, surfaces may be represented using a local binarypatterns (LBP) algorithm. Using the LBP algorithm, the touch controller112 detects 10 microstructures inside the texture of the surface shownin an image. The 10 microstructures are features that can be used todistinguish one surface from another. In other embodiments, the touchcontroller 112 may extract features from the colors in an image of asurface.

To better identify surfaces, the touch controller 112 may include aclassifier. Persons skilled in the art will recognize that a variety oftechniques may be used to implement a classifier. For example, in oneembodiment, the classifier may be implemented as the library for supportvector machine (LIBSVM). The touch controller 112 trains the classifierwith the library. The library includes images and features of particulartypes of surfaces, e.g. the surface of a desk. The touch controller 112may present an interface through which the user can add new surfacerepresentations and associated images or update the library withadditional images for an existing surface. Once trained, the touchcontroller 112 may use the classifier to identify a surface, based uponthe features in an image.

As noted, the touch controller 112 is also configured to recognizefiducial markers (e.g. data matrix codes) within an image. Todistinguish fiducial markers, the touch controller 112 may include adecoder. For example, in one embodiment, the decoder may be implementedwith the data matrix decoding package icEveryCode.™ The decoderdetermines whether a fiducial marker, such as a barcode, is present. Ifa fiducial marker is present, then the decoder determines a valueassociated with the fiducial marker. The touch controller 112 may storevarious fiducial markers within the library.

The touch controller 112 allows a user to configure a mapping from atouch on a given surface to a specified action. That is, the touchcontroller 112 allows a user to configure how the computer system 110should respond to touches made by a user. The touch controller 112 maypresent an interface that includes a list of surfaces and a list ofactions. The list of surfaces may include sample images and usersupplied labels for each surface. When the user selects an action for aparticular surface, the touch controller 112 maps the selected action tothe selected surface.

As discussed, the touch controller 112 may store mappings defined by theuser in the library. For example, if the user selects the action ofmuting the ringtone of a phone when the user touches the surface of ashirt, then the touch controller 112 would store a mapping between theaction of muting the ringtone and the representation of the shirt in thelibrary. Likewise, the user could map a surface, such as an icon made ofthe text characters, to the action of launching an application. Then theuser can place the icon in a convenient location and launch theapplication by tapping the device 120 on the icon.

Once actions are mapped to surfaces, the touch controller 112 canexecute an action when the user touches a surface. The touch controller112 executes an action, in response to receiving images of a touchedsurface from the device 120. To determine what action to invoke, thetouch controller 112 identifies the touched surface based on an imagereceived from the device 120. The touch controller 112 identifies thetouched surface by matching an image of the touched surface from thedevice 120 to the library. As discussed, the touch controller 112 mayuse a classifier to match an image to a surface representation in thelibrary. In some cases, the touch controller 112 uses the LBP algorithmto detect texture features in the image. The classifier then matches thetexture features of the image to texture features of the representationof a particular type of surface in the library. Once identified, thetouch controller 112 executes the action associated with the surfacerepresentation. That is, the touch controller 112 executes an action inresponse to the user touching the finger 115 on a surface.

While described as including the library of surface representations, inother embodiments, the computer system 110 may access, via network 120,surface representations stored on a physical computing system (e.g., asystem in a data center) or a virtual computing instance executingwithin a computing cloud.

In addition, the touch controller 112 may execute an action based on agesture made on a given surface. In one embodiment, the device 120 mayinclude an optical flow sensor. The optical flow sensor evaluates imagesof the surface to determine a direction and a speed of movement, andtherefore the direction and speed of movement of a user performing agesture.

To recognize gestures, the touch controller 112 receives data describingthe direction and speed of movement of the device 120, the touchcontroller 112 compares the data against patterns of movement toidentify the gesture made by the user. The user maps actions to gesturesmade on surfaces using the touch controller 112. That is, a surfacerepresentation may be associated with mappings, distinguished bygesture, such that the touch controller 112 may select an action basedon (1) a surface and (2) a gesture performed on that surface.

While described as executing commands, in other embodiments, the actionsmay provide continuous control of various parameters. For example, touchcontroller 112 could turn down the volume of an audio speaker when theuser swipes the device 120 down a side of the audio speaker. The touchcontroller 112 could also control a pointer on a screen displayed bycomputer system 110 when the user gestures on the back of a tabletcomputer. Controlling the pointer with gestures on the back of a tabletcomputer allows the user to interact with the tablet computer withoutoccluding the screen.

In still other embodiments, the actions may change between differentoperating modes. For example, when in a normal mode the touch controller112 could launch a word processing application if the user taps on atable. However, if the user taps on a print-out of a presentation, thetouch controller 112 could enter a presentation mode. If the user tapson a table while the touch controller 112 is in this presentation mode,then the touch controller 112 could couple the display of the computingdevice 110 to a projector. If the user then pinches the device 120against their thumb while the touch controller 112 remains inpresentation mode, then the touch controller 112 could advance thepresentation. That is, the current operating context of an applicationmay be used to take different actions for the same touched surface (orsurface and gesture).

While described as mapping single surfaces and gestures to actions, inother embodiments, the touch controller 112 may map combinations ofmultiple gestures and surfaces to actions. For example, the touchcontroller 112 could display a presentation via a projector, if the userswipes from a print-out of the presentation to a table.

In addition, in another embodiment, the touch controller 112 may allowthe user to define custom gestures, e.g. swiping vertically and thenswiping horizontally to form a cross or dragging the device 120 in acircle. The user maps actions to these custom gestures made on surfacesusing the touch controller 112.

FIG. 2 illustrates an example of a device 120 used to instrument afinger, according to one embodiment. The device 120 captures images of asurface and the motion of a finger proximate to the surface. Oncecaptured, the device 120 transmits the images to a computing system thatinvokes an action based on the surface touched by a user. The device 120can be provided in a variety of form factors that can fit on a finger.For instance, the device 120 could be embedded on a ring or thimble likestructure worn on the tip of the finger. Doing so allows the device 120to be available, but unobtrusive to the user. This fitting also allows auser to remove the device 120 when desired, or twist the device 120 todeactivate sensing of touched surfaces. Alternatively, the device 120may be embedded under a user's fingernail, on the surface of thefingertip, or partially implanted under the skin of the finger withexposed components for sensing.

As shown, the device 120 includes a microcontroller 206 coupled to apower supply 208, a camera 202, an optical flow sensor 210, and alight-emitting diode (LED) 204. A communications link 105 couples thedevice 120 to the computer system 110. As discussed, communications link105 may include a wired connection that transports data and power, suchas a USB connection.

The power supply 208 is configured to distribute power to the variouscomponents of the device 120. The power supply 208 may receive powerfrom the computer system 110 via communications link 105. For instance,the communications link 105 may include a USB connection that carriespower. In other embodiments, the power supply 208 may produce or storepower. For instance, the power supply 208 may include a rechargeablebattery. The power supply 208 may also include circuitry to harvestambient power from the surrounding environment, e.g. the body or motionof the user.

In one embodiment, the device 120 captures images of a surface withcamera 202. In one embodiment, the camera 202 may be a micro red greenblue (RGB) camera, e.g. the AWAIBA NanEye micro RGB camera. The smallform factor of camera 202 allows it to fit in the device 120. The camera202 captures an image in response to receiving a signal from themicrocontroller 206. The camera 202 can provide an image to themicrocontroller 206 as collections of pixels, e.g., a 248×248 pixelimage. Depending on the position of the camera 202 relative to othercomponents of the device 120, the borders of the captured images mayinclude artifacts, e.g., shadows. As discussed below, the touchcontroller 112 may crop images captured by the camera 202 to remove suchartifacts.

In one embodiment, the device 120 captures gestures with the opticalflow sensor 210. As discussed, the optical flow sensor 210 is configuredto detect motion across a surface. The optical flow sensor 210 mayinclude a high-speed but low-resolution camera. In one embodiment, theoptical flow sensor 210 may be an ADNS 2620 optical flow sensor,commonly used in optical mice. When proximate to a surface, the opticalflow sensor 210 detects motion by rapidly capturing images of thesurface, identifying differences between the images, and calculating adirection and speed of movement based upon the differences. The opticalflow sensor 210 may transmit the direction and speed of movement as aseries of coordinates, using an initial point of contact with thedevice. After the initial point of contact, coordinates are providedthat indicate changes in position relative to the initial point ofcontact.

In other embodiments, a mechanical device (e.g. a trackball) or anaccelerometer may be used in place of the optical flow sensor 210.Although described as distinct components, in still other embodiments,the camera 202 and optical flow sensor 210 may be combined into a singlecomponent with a camera capable of capturing high-resolution images athigh-speeds.

When the finger is pressed against a surface, the ambient lighting maybe insufficient for the camera 202 and optical flow sensor 210 tocapture images. Accordingly, the LED 204 provides light for the camera202 and optical flow sensor 210. The camera 202, optical flow sensor210, and LED 204 are positioned proximate to one another in the device120.

The microcontroller 206 is configured to control the operation of thecamera 202, optical flow sensor 210, and LED 204. The microcontroller206 also retrieves data from the camera 202 and optical flow sensor 210.The microcontroller 206 may process the data retrieved from the camera202 and optical flow sensor 210.

For instance, the microcontroller 206 may process images from the camera202 to determine when the device 120 is contacting a surface. To do so,the microcontroller 206 continually retrieves images from the camera202. The microcontroller 206 determines whether the device 120 iscontacting a surface by identifying changes in contrast betweensubsequent images. The microcontroller 206 may determine the contrastwithin a region of the image by averaging a square difference betweeneach pixel and a neighboring pixel in the region. When the contrast ofan image is more than twice the contrast of the previous image, themicrocontroller 206 determines that the device 120 initially contacts asurface. When the device 120 contacts a surface, the microcontroller 20fs6 continues to analyze the contrast. The microcontroller 206determines that the device 120 is no longer contacting a surface whenthe contrast of the current image is less than a threshold value.Alternatively, the device 120 may include a switch that activates whenthe device 120 touches a surface.

While the device 120 contacts a surface, the microcontroller 206transmits images (and gesture data) to the computer system 110. Asdiscussed, the computer system 110 analyzes this information todetermine which action (if any) to execute. Thus, the device 120initiates actions by detecting contact with a surface and transmittingimages to the computer system 110.

Although discussed as transmitting data to the computer system 110, inother embodiments, the device 120 may also receive instructions or datafrom the computer system 110. The computer system 110 may instruct themicrocontroller 206 to continually retrieve and transmit image data,whether the device 120 is contacting a surface or not. For instance, thedevice 120 could enable a user to look under an object by continuallyretrieving and transmitting image data while the user holds their fingerunder the object. In such a case, the computer system 110 would displaythe image data to the user.

In other embodiments, the device 120 may communicate with other devices.For instance, the device 120 may send data encoded in Morse code byblinking the LED 204. The device 120 may also receive data byidentifying a pattern of blinking light in the images that the camera202 captures.

FIG. 3 illustrates regions of an image 300 of a surface, according toone embodiment. The camera 202 captures the image 300, which themicrocontroller 206 and touch controller 112 analyze. Instead ofanalyzing the entire image, the microcontroller 206 and touch controller112 may more efficiently analyze regions within the image. As shown, theimage 300 includes two such regions, illustrated as a field of view 302and a contrast region 304.

As discussed, the microcontroller 206 analyzes images captured by thecamera 202 to determine whether the device 120 has contacted a surface.The microcontroller 206 may optimize this determination by analyzing thecontrast region 304 of each image, instead of the entire image. If theimage is 248×248 pixels, then the contrast region 304 may be 60×60pixels. Since the contrast region 304 includes fewer pixels, themicrocontroller 206 can perform an analysis more efficiently.

In other embodiments, the microcontroller 206 may further optimize thedetermination of whether the device 120 is contacting a surface, byconverting color images from the camera 202 to grayscale images. Themicrocontroller 206 may perform this conversion because grayscale imagesare typically faster to analyze than color images. The microcontroller206 may also transmit the smaller grayscale images to the touchcontroller 112 instead of the color images.

The touch controller 112 selects an action to execute when the usertouches a surface, by identifying the touched surface. The touchcontroller 112 identifies the touched surface by matching images of thetouched surface from the device 120 to a library. As discussed, thetouch controller 112 matches an image to the library according to thefeatures in the image. However, depending on the position of the camera202 and other components within the device 120, the edges of the image300 may include various artifacts (e.g. shadows or wires). Theseartifacts may prevent the touch controller 112 from accuratelyidentifying the surface shown in an image. Therefore, the touchcontroller 112 may analyze the features shown within the field of view302 instead of the entire image 300.

FIG. 4 illustrates a method for determining when a finger device iscontacting a surface, according to one embodiment. Although the methodsteps are described in conjunction with the system of FIG. 1 and FIG. 2,persons skilled in the art will understand that any system configured toperform the method steps, in any order, is within the scope of thepresent invention.

As shown, method 400 begins at step 405, where the microcontroller 206retrieves a current image from the camera 202. As noted, the imageitself may comprise an array of pixel values.

At step 410, the microcontroller 206 determines a contrast of thecurrent image. As discussed, the microcontroller 206 may determine thecontrast by averaging the squared difference between each pixel and aneighboring pixel within a region of the current image. Themicrocontroller 206 may determine the contrast within the contrastregion 304.

At step 415, the microcontroller 206 determines if the device 120 iscontacting a surface. To determine when the device 120 initiallycontacts a surface, the microcontroller 206 compares the contrast of thecurrent image to the contrast of the previous image. The microcontroller206 continually retrieves images and calculates the contrast for theimages. The microcontroller 206 also stores the contrast of the previousimage for comparison. If the microcontroller 206 does not yet have thecontrast of the previous image stored, then the microcontroller 206determines that the device 120 is not contacting a surface. If thecontrast of the current image is less than or equal to twice thecontrast of the previous image, then the device 120 is not contacting asurface. However, if the contrast of the current image is more thantwice the contrast of the previous image, then the device 120 iscontacting a surface.

While the device 120 remains in contact with a surface, themicrocontroller 206 compares the contrast of the current image to athreshold value. If the device 120 has been contacting a surface and thecontrast is less than the threshold value, then the device 120 is nolonger contacting the surface. If the device 120 has been contacting asurface and the contrast is greater than or equal to the thresholdvalue, then the device 120 is still contacting the surface. If themicrocontroller 206 determines that the device 120 is not in contactwith a surface, then the method 400 returns to step 405. Otherwise, themicrocontroller 206 determines that the device 120 is in contact asurface and the method 400 proceeds to step 420.

At step 420, the microcontroller 206 determines movement of the device120 relative to the surface. As noted, an optical flow sensor 210 may beused to track the movement of the device 120 across a surface. In thiscase, the microcontroller 206 retrieves coordinates representing themovement of the device 120 from the optical flow sensor 210.

At step 425, the microcontroller 206 transmits image and coordinates tothe computer system 110. While the device 120 is contacting a surface,the microcontroller 206 continues to retrieve and transmit images to thecomputer system 110. The series of coordinates transmitted by themicrocontroller represents the gesture of the finger on a surface.

FIG. 5 illustrates a method for associating an action to surface touchesmade by a user wearing an instrumented figure device, according to oneembodiment of the present invention. Although the method steps aredescribed in conjunction with the system of FIG. 1 and FIG. 2, personsskilled in the art will understand that any system configured to performthe method steps, in any order, is within the scope of the presentinvention.

As shown, method 500 begins at step 505, where the touch controller 112receives a training or reference image of a surface from the device 120.For example, a user may touch the surface of their desk with the device120. The device 120 would then capture an image of the desk and transmitthat image to the touch controller 112 as a training image.

At step 510, the touch controller 112 displays the training image and alist of actions. At step 515, the touch controller 112 receives inputspecifying an action to associate with the surface shown in the trainingimage. In addition, the user may also specify a gesture required for theaction. For instance, if the user would like a word processor to launcheach time the device 120 swipes the displayed surface, then the userwould select the swipe gesture and the action of launching the wordprocessor.

At step 520, the touch controller 112 maps a surface representation tothe specified action. The touch controller 112 determines features ofthe surface shown in the training image. The touch controller 112 maydetermine features from the texture of the surface shown in the trainingimage. The touch controller 112 then adds the texture features, trainingimage, and specified action to a library. As discussed, the touchcontroller 112 may include a classifier. The touch controller 112 maytrain the classifier on the texture features extracted from the trainingimage. Doing so trains the classifier to identify a surfacerepresentation (and associated action) when the touch controller 112receives subsequent images of the surface from the device 120.

In addition, the touch controller 112 may associate various actions togestures on a surface. The touch controller 112 may add the gesture tothe library. As such, when the touch controller 112 identifies a surfacerepresentation, the touch controller 112 may further distinguish anaction to execute based upon a gesture received from the device 120. Forexample, tapping on the surface of a desk could be mapped to the actionof opening a word processor, but the swiping across the desk could bemapped to saving a document that is open in the work processor.

FIG. 6 illustrates a method for initiating an action based on a usertouching a surface with an instrumented finger device, according to oneembodiment. Although described in conjunction with the system of FIG. 1and FIG. 2, persons skilled in the art will understand that any systemconfigured to perform the method steps, in any order, is within thescope of the present invention.

As shown, method 600 begins at step 605, where the touch controller 112receives an image when the device 120 contacts a surface. The imageincludes sufficient detail for the touch controller 112 to determinefeatures of the texture of the surface. As discussed, the touchcontroller 112 may also receive coordinates representing the motion ofthe device 120 on the surface.

At step 610, the touch controller 112 matches the image to a surfacerepresentation in a library. As discussed, the touch controller 112 mayextract texture features from the image. The touch controller 112 maythen use a classifier to identify a surface representation in thelibrary, based on the extracted texture features. As noted, the imagemay include fiducial markers. Accordingly, the touch controller 112 mayincludes a decoder capable of recognizing a fiducial marker. Oncerecognized, the touch controller 112 identifies a surface representationbased on the fiducial marker.

At step 615, the touch controller 112 determines if an action isassociated with the surface representation. If there is not an actionassociated with the surface representation, then the method 600 ends.Otherwise, if the is an action associated with the surfacerepresentation, then at step 620, the touch controller 112 executes theassociated action. The associated action may include a command (e.g.launching an application or sending a notification), or a variety ofother programmatic responses to a user touching the surface.

As noted above, the surface may be associated with actions,distinguished by gestures. The touch controller 112 may thereforedetermine a gesture from coordinates received in step 605. Afterdetermining the gesture, the touch controller determines if an action isassociated with the gesture on the surface.

FIG. 7 illustrates a computing system configured to implement one ormore aspects of the present invention. As shown, the computing system110 includes, without limitation, a central processing unit (CPU) 760, anetwork interface 750 coupled to a network 755, a memory 720, andstorage 730, each connected to an interconnect (bus) 740. The computingsystem 110 may also include an I/O device interface 770 connecting I/Odevices 775 (e.g., keyboard, display, mouse, three-dimensional (3D)scanner, and/or touchscreen) to the computing system 110. Further, incontext of this disclosure, the computing elements shown in computingsystem 110 may correspond to a physical computing system (e.g., a systemin a data center) or may be a virtual computing instance executingwithin a computing cloud.

The CPU 760 retrieves and executes programming instructions stored inthe memory 720 as well as stores and retrieves application data residingin the storage 730. The interconnect 740 is used to transmit programminginstructions and application data between the CPU 760, I/O devicesinterface 770, storage 730, network interface 750, and memory 720. Note,CPU 760 is included to be representative of a single CPU, multiple CPUs,a single CPU having multiple processing cores, and the like. And thememory 720 is generally included to be representative of a random accessmemory. The storage 730 may be a disk drive storage device. Althoughshown as a single unit, the storage 730 may be a combination of fixedand/or removable storage devices, such as fixed disc drives, removablememory cards, or optical storage, network attached storage (NAS), or astorage area-network (SAN).

Illustratively, the memory 720 includes the touch controller 112,various surface representations 724, actions 726, and mappings 728. Thevarious surface representations 724 may be included within a library,not shown. The touch controller 112 includes a configuration tool 722that creates the mappings 728 between the surface representations 724and actions 726. The configuration tool 722 may create the mappingsbased upon user input. As discussed, the configuration tool 722 may mapmultiple actions, such as action 726-2 and 726-3 to a single surfacerepresentation 724-2. The various mappings may be associated withgestures or modes of the touch controller 112. Thus, the touchcontroller 112 identifies an action to execute based upon the surfacethat the user contacts, the gesture made on the surface, and the mode ofthe touch controller 112.

The storage 730 includes various applications 732. An action may includecommands to launch an application. For instance, the action 726-1 mayinclude commands to launch application 732-2. The touch controller 112,thereby, launches the application 732-2 in response to the userinteracting with surface 724-1, which is mapped to action 726-1.

One embodiment of the invention may be implemented as a program productfor use with a computer system. The program(s) of the program productdefine functions of the embodiments (including the methods describedherein) and can be contained on a variety of computer-readable storagemedia. Illustrative computer-readable storage media include, but are notlimited to: (i) non-writable storage media (e.g., read-only memorydevices within a computer such as CD-ROM disks readable by a CD-ROMdrive, flash memory, ROM chips or any type of solid-state non-volatilesemiconductor memory) on which information is permanently stored; and(ii) writable storage media (e.g., floppy disks within a diskette driveor hard-disk drive or any type of solid-state random-accesssemiconductor memory) on which alterable information is stored.

The invention has been described above with reference to specificembodiments. Persons skilled in the art, however, will understand thatvarious modifications and changes may be made thereto without departingfrom the broader spirit and scope of the invention as set forth in theappended claims. The foregoing description and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense.

We claim:
 1. A device worn on a finger, the device comprising: a cameraconfigured to capture images of a surface; and a microcontroller thatdetects that the finger of a user wearing the device touches a surfaceand, in response, (i) captures an image of the surface and (ii)transmits the image to a computing system.
 2. The device of claim 1,wherein the microcontroller further measures a relative movement of thefinger device while in contact with the surface.
 3. The device of claim1, wherein detecting that the finger of a user wearing the devicetouches a surface comprises: calculating a first contrast associatedwith a first image retrieved by the microcontroller; calculating asecond contrast associated with a second image retrieved by themicrocontroller; and determining that a difference value associated withthe first contrast and the second contrast exceeds a threshold value.